Papers with shuffling and packing methods
Optimizing Packing and Shuffling Strategies for Enhanced Performance in Generative Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Documents are typically concatenated to chunks of maximum sequence length (MSL) and shuffled in chunks (atom-size chunks). |
| Approach: | They propose to pack and shuff documents in chunks of tokens to prevent overfitting . they also propose to use padding to only include one document per chunk . |
| Outcome: | The proposed method reduces the risk of overfitting and improves generalizability. |